1. Field of the Invention
The present invention relates to a technology of performing image processing, more specifically to an image processing apparatus, an image processing method, a storage medium, and an image processing system that project a three-dimensional image photographed with a relatively wide view angle to a planar image.
2. Description of the Related Art
A system that displays a still image photographed by, for example, a camera on a flat display device connected to a computer has been known. The computer of the system combines a plurality of images of a subject photographed from different angles in a partially overlapping manner and causes an image viewer operated in the computer to display, on a flat display device, a composite image combining the plurality of images (hereinafter, the system will be referred to as a panoramic image viewer).
The existing panoramic image viewer supports various display changing operations, such as panning (i.e., leftward or rightward movement of the field of view), tilting (i.e., upward or downward movement of the field of view), and zooming (i.e., magnification or reduction), to be performed on a panoramic image combining the plurality of images. The panoramic image is formed as a combination of images of the same subject photographed at different angles. Therefore, particularly when images photographed by a camera having a zoom function or a fisheye lens for photographing in a wider field of view than the field of view of a normal lens system are projected onto a plane and combined together, a person may perceive unnaturalness or distortion in the panoramic image particularly at edge portions of the field of view.
The panoramic image viewer typically employs a method of projecting and displaying, on a plane, an image disposed on a cylindrical side surface of a cylinder or a spherical surface of a sphere and as viewed from the center of the cylinder or the sphere. When projecting and displaying such an image on a plane, a user is allowed to perform processes, such as panning, tilting, and zooming, on the image to be displayed. Further, a three-dimensional image focused on the cylindrical side surface of the cylinder or the spherical surface of the sphere is projection-transformed to a planar image and displayed on a flat display in accordance with user-set pan, tilt, and zoom parameters.
In this case, a central projection method, which is a projection transformation method used for a standard camera lens, is typically used. According to the method, the image displayed on the plane looks similar to the image photographed by a camera.
For example, to transform a three-dimensional image acquired in an omnidirectional format to an image on a flat display to be viewed by a person, projection transformation is performed to transform the three-dimensional image to a planar image. In the projection transformation, on the assumption that an image disposed on the spherical surface of a sphere is viewed from the center of the sphere, the angle of a given point of the image from the center (i.e., half view angle) is transformed to the distance of the point from the center of a plane (i.e., image height).
As well as the above-described central projection method, major projection transformation methods include a stereographic projection method, an equidistant projection method, an equisolid angle projection method, and an orthogonal projection method. FIG. 1 illustrates respective projection transformation equations and inverse projection transformation equations of projection transformation methods used in the above-described projection methods. The central projection method is usually employed in photographing by a camera equipped with a standard camera lens. The other four methods are usually employed in photographing by a camera with a fisheye lens having a super-wide view angle.
Each of FIGS. 2A to 2D illustrates an image photographed by a camera and mapped onto a plane. FIG. 2A illustrates an image photographed with a lens system having a relatively narrow view angle and mapped onto a plane by the central projection method. FIG. 2B illustrates an image photographed with a lens system having a relatively wide view angle and mapped onto a plane by the central projection method. FIG. 2C illustrates an image photographed with a lens system having a relatively narrow view angle and mapped onto a plane by the stereographic projection method. FIG. 2D illustrates an image photographed with a lens system having a relatively wide view angle and mapped onto a plane by the stereographic projection method. In many image processing apparatuses each functioning as the panoramic image viewer, the projection method is fixed to the central projection method, not depending on the maximum half view angle of the image to be displayed. This is because the maximum half view angle of a camera used in the existing panoramic image viewer is approximately 30 degrees, and visually obvious image distortion is barely perceived within this half view angle.
Further with reference to FIGS. 2A to 2D, if the view angle is relatively narrow, the image based on the central projection method is not particularly unnatural, as illustrated in FIG. 2A. If a camera having a relatively wide view angle is used, however, an issue arises. For example, as illustrated in FIG. 2B, if the central projection method is applied to the image photographed with a relatively wide view angle, an unnatural image is produced in which a person, who is the same subject as that of FIG. 2A, has an arm stretched unnaturally long toward an edge portion of the image.
That is, when a three-dimensional image photographed with a relatively wide view angle is projected to a planar image in accordance with the central projection method, visual unnaturalness is produced in an area of a relatively wide half view angle, i.e., in an edge portion of the planar image, as illustrated in FIG. 2B. In addition, as understood from the transformation equations illustrated in FIG. 1, according to the central projection method, the value overflows when a half view angle φ is 90 degrees. To display an area of a maximum half view angle of 90 degrees or more, therefore, an additional process is required.
Meanwhile, FIGS. 2C and 2D illustrate planar images obtained from the same images as those used in FIGS. 2A and 2B through transformation using the stereographic projection method as a different projection method. FIG. 2C illustrates an image photographed with a lens system having a relatively narrow view angle, and FIG. 2D illustrates an image photographed with a lens system having a relatively wide view angle. As illustrated in FIGS. 2C and 2D, it is understood that the stereographic projection reduces the unnaturalness, i.e., the unnaturally stretched arm of a person, observed in one of the planar images according to the central projection method at an edge portion of the planar image corresponding to image data of the three-dimensional image having a relatively wide half view angle.
If the above-described images are examined more closely, however, it is recognized that the images of background lines appear substantially straight in the images according to the central projection, but appear slightly bent in the images according to the stereographic projection in both the relatively narrow view angle and the relatively wide view angle. As described above, the existing panoramic image viewer is limited in providing an image with no unnaturalness produced in an area of a relatively wide view angle.
In many standard cameras, the view angle ranges from approximately 50 degrees to approximately 75 degrees. As illustrated in FIG. 2B, if an image photographed with a relatively wide view angle is transformed to a planar image only by the use of the central projection method, the planar image has noticeable deformation attributed to substantial stretching of the image occurring in edge portions of the image having a relatively wide view angle, in which the image is substantially stretched in the tangential direction of a polar coordinate system centering on a point of the image corresponding to the center of the lens. Consequently, if a system directly using the existing projection transformation method is employed as the panoramic image viewer using a still image photographed with a relatively wide view angle, more or less unnaturalness is produced in the entire resultant image.
Concerning the above-described projection methods, there is a known technique of projecting onto a plane an arbitrary portion of a distorted circular image photographed with a fisheye lens. According to the background technique, to cut out the desired arbitrary portion from the distorted circular image photographed with a fisheye lens and transform the cut-out image to a less distorted planar image, the image photographed with a fisheye lens and related to a spherical surface of a sphere is cut out to be related to a cylindrical side surface of a cylinder having an axial direction perpendicular to the planar image, and thereby the distortion occurring at horizontal edges of the image is adjusted.
The above-described technique is capable of reducing visually perceptible unnatural distortion; however, it reduces the unnatural distortion by using a curved surface of the cylinder, and thus the reduction of the unnatural distortion is limited to one direction, i.e., the horizontal direction. Therefore, a planar image distorted differently between the horizontal direction and the vertical direction may be produced, and thus visual unnaturalness different from that caused by the foregoing existing panoramic image viewer may be caused. That is, in the panoramic image viewer that displays, on a substantially flat display surface, an image photographed with a relatively wide view angle, the reduction of unnaturalness of the image, which is attributed to image distortion caused by stretching of a subject at upper, lower, left, and right edge portions of the image, still remains to be addressed.